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Abstract 

Methods for understanding classical disordered spin systems with interactions conforming 
to some idealized graphical structure are well developed. The equilibrium properties of the 
Sherrington-Kirkpatrick model, which has a densely connected structure, have become well un- 
derstood. Many features generalize to sparse Erdos-Renyi graph structures above the percolation 
threshold, and to Bethe lattices when appropriate boundary conditions apply. In this paper we 
consider spin states subject to a combination of sparse strong interactions with weak dense inter- 
actions, which we term a composite model. The equilibrium properties are examined through the 
replica method, with exact analysis of the high temperature paramagnetic, spin glass and ferro- 
magnetic phases by perturbative schemes. We present results of a replica symmetric variational 
approximations where perturbative approaches fail at lower temperature. Results demonstrate 
novel reentrant behaviors from spin glass to ferromagnetic phases as temperature is lowered, in- 
cluding transitions from replica symmetry broken to replica symmetric phases. The nature of high 
temperature transitions is found to be sensitive to the connectivity profile in the sparse sub-graph, 
with regular connectivity a discontinuous transition from the paramagnetic to ferromagnetic phases 
is apparent. 
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I. INTRODUCTION 



Statistical physics methods for studying disordered spin systems have become well de- 
veloped. Much of the development can be traced back to early work on mean-field models 
for disordered magnetic systems and the theory was strongly developed in spin-glass mod- 
els [1, 2]. One problem in studying spin glasses and disordered media has been in appropri- 
ately modeling the inhomogeneity within tractable frameworks. Statistical descriptions of 
inhomogeneity are often realized by random coupling ensembles. Small systems described 
in this way may have strongly varying properties, but the ensemble may be chosen so that 
the macroscopic description is asymptotically well defined. 

Both dense and sparse graphical models are useful in understanding a range of phenom- 
ena, such as neural networks [3] , information theory [4] and other information processing [5] , 
where spatial and dimensional constraints are often less rigid. Many complex systems have 
an inhomogeneous interaction structure that can be approached, if not exactly represented, 
by consideration of simple random graph ensembles. In this paper, spin glass models with 
couplings conforming to infinite dimensional Erdos-Renyi random graphs are considered [6] . 
In the large system limit many equilibrium properties depend on the connectivity distribu- 
tion, and how the number of couplings per variable scales with N, the system size. Dense 
graphs have a number of links per variable that is O(N) in the large system limit, whereas 
sparse ensembles have finite mean connectivity in this limit. Many topological features be- 
come well defined in these limits. Two standard sparse coupling distributions are considered, 
a description with regular user connectivity, and one with Poissonian user connectivity. The 
distinctions between these two sparse models and the limiting case of full connectivity are 
illustrated in figure 1. 

Some densely connected models may be analyzed exactly for ensembles of uniform bi- 
nary interactions, and certain random coupling models, most famously the Sherrington- 
Kirkpatrick (SK) model of spin glasses [7]. Simplification of the analysis in the disordered 
case is often possible through noting the ability to describe large sets of interactions by 
central limit theorems [8]. For sparse connectivity models, such as the Viana-Bray (VB) 
model [9, 10], a locally tree like approximation (Bethe approximation) is often essential in 
simplifying analysis; central limit theorems again apply to certain objects, but not directly 
to the set of local interactions for any variable. Models which do not allow use of central 
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limit theorems or locally tree-like approximations are normally significantly more difficult 
to analyze. 




FIG. 1: (color online) Shown are the couplings (links) amongst a set of spin variables (circles), 
which describes graphically a particular quadratic Hamiltonian. Left figure: The fully connected 
graph is a special case of a dense graph describing the SK model, with O(N) non-zero couplings 
per variable in the large system limit. Centre/Right figure: The VB model is defined with O(l) 
non-zero couplings per variable in the large system limit. Center figure: In the DS - removed ',' case 
of a regular connectivity random graph above the percolation threshold there is an inhomogeneous 
structure on a global scale, but locally the structure is a Bethe lattice (regular tree). Right figure: 
In the case of a random graph with Poissonian connectivity the local structure is again tree like. 
Above the percolation threshold many trees of finite size, and unconnected variables exist, as well 
as a giant component containing O(N) variables, and many loops [6]. The 1-core contains all 
variables with at least one link, including the giant component above the percolation threshold. 
Additional structures within the giant component may be identified, including a 2-core, obtained 
by recursively removing leaves (singly connected variables) from the giant component. 

Frameworks in which an interplay between strong sparse and weak pervasive couplings 
might be proposed in a variety of areas. In nanotechnology for example, miniaturization of 
classical components will preserve engineered short range interactions, but other accidental 
correlations may emerge not limited by the designed connectivity structure, and these may 
well be modeled by a mean-field (infinite connectivity) like interaction. A mixed connectivity 
may also be a designed feature. Neuronal activity is known to involve a combination short 
and long range information processing structures, this motivated a 1 + oo dimensional model 
of neuronal activity [11] discovering many novel properties. Another example of such an 
engineering application is CDMA, where improvement over standard methods is possible [12]. 
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To motivate a closely related study, Hase and Mendes noted a possible application for 
theories of these structures [13]: Consider the model with sparse anti-ferromagnetic couplings 
on a structure otherwise fully connected through ferromagnetic couplings. This composite 
model can be considered as one in which a ferromagnetic phase is maintained by a densely 
connected network, but with a small proportion of links attacked. Often only a small portion 
of a link structure is accessible to an attacker, so it is interesting to consider how the system 
response differs from weak attacks on all (or most) links. 

The effect of an attack on a sparse subset may cause a transition away from the ordered 
phase, when sufficiently strong. It is possible that the nature of transitions away from 
the ordered state may differ from those with only a single interaction scale. The effect of 
disruption of networks by random attack, or frustrating interactions, is of importance in 
many practical network models [13, 14], the restriction to random topologies allows a focus 
on generic properties, in this case restricted to the issue of sparse and dense induced effects. 

More generally, a range of mean field behavior, including spin glass like, may be supported 
by the dense sub-structure, combined with an arbitrary set supported by the sparse sub- 
structure, as shown in figure 2. In so doing, a wider variety of competitive phase behavior 
are explored. 




FIG. 2: (color online) Left figure: A sparse model defined by some mean connectivity, describes 
couplings in the sparse model. Center figure: A fully connected model, describes couplings in the 
dense model. Right figure: A fully connected graph with a subset of strong sparse links, this is 
the composite model. The sparse subset of couplings are an order of magnitude stronger than the 
couplings on the other edges. 

It may be expected that many of the results for composite systems will be similar to those 
for the limiting sparse and dense models. Four thermodynamic phases describe equilibrium 
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properties of spin models with independent and identically distributed (i.i.d) couplings. A 
pure state with no macroscopic order, the paramagnetic phase; a pure state with macro- 
scopic order aligned with some mean bias in the couplings, the replica symmetric (RS) 
ferromagnetic phase (F); a macroscopically aligned phase, but with some complicated phase 
space fragmentation, the mixed phase (M); and a phase with no macroscopic alignment and 
a complicated fragmentation of the phase space, the spin glass phase (SG). Within both the 
sparse and dense Ising spin models these phases are exhibited and many features are shared 
by the two models. 

The main question investigated in this paper is how phase behavior and transitions differ 
in the composite model from the sparse and dense frameworks, and whether a simple inter- 
polation is produced by the composite models. Attention is restricted to cases in which the 
sparse sub-structure is percolating, since in any other regime the long range coupling will 
be due solely to the dense links. A non-percolating sparse substructure would not test the 
effects of competing long range induced order, although some of the methods and results 
are inclusive of this scenario and appear to vary continuously (at finite temperature) across 
the percolation threshold corresponding to the sparse substructure. 

Section II outlines the ensemble of models studied, which are then analyzed by the replica 
method in section III. Section IV develops the replica equilibrium solution by population 
dynamics alongside an analogous belief propagation (BP) algorithmic method applicable to 
typical samples. A stability analysis of the BP equations and population dynamics is derived. 
Section V presents a leading order solution to the composite system in terms of a simplified 
ansatz on the order parameter with results discussed in section VI. Section VII demonstrates 
the RS solutions for several composite models in the interesting range of parameters about 
the triple point in the phase diagram. Section VIII presents hypotheses on the structure of 
the low temperature phases alongside experimental results derived by BP and monte-carlo 
methods. We then conclude with a brief summary. 

II. COMPOSITE ENSEMBLES 

The Composite model can be described by a Hamiltonian with coupling of N spins 




(1) 
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where (ij) are an ordered set of variables. The couplings are labeled as dense (D) or sparse 
(S) and are sampled independently for each link according independent ensembles described 
shortly. The quenched variable abbreviation Q indicates a sample of the couplings, and 

— * 

S are the dynamic variables. The field vector z is used only as a conjugate parameter to 
explore symmetries, the limit z — > (vector of zero fields) is always assumed throughout 
this chapter, although some physical quantities and insight are demonstrated using conjugate 
fields as described in Appendix B. 

The equilibrium properties of the model are studied. The Hamiltonian implies a static 
probability distribution on the state space given by 

where (3 is the inverse temperature and Z is the partition function. 

The spin states of interest are the typical case equilibrium distribution, in the large system 
limit. Properties of these states are established through the mean free energy 

Mb (/?) = - lim ^(logZ) s , (3) 

N^oo iv 

where S is the ensemble parameterization. 

The model is fundamentally a fully connected one, the sparse component is realized as a 
subset of couplings that are an order of magnitude stronger. Due to this order of magnitude 
many results for standard densely connected spin models do not apply. 

1. Dense (SK) sub-structure 

The dense sub-structure fully connects N spin variables S G {±1}^, with couplings 
sampled independently and at random according to the Gaussian distribution parameterized 
by J and J 




with a necessary scaling of components included. This set of couplings has a statistical 
description corresponding to the SK model. 
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2. Sparse (VB) sub-structure 



It is convenient to factorize the sparse couplings as 

Jfij) = A (ij) v (ij) ■ ( 5 ) 

The ensemble is described by a connectivity matrix, A, which is zero for all but a fraction 
C/N of components, and a dense coupling matrix V, with no zero elements. In the irregular 
ensemble each directed edge is present (non-zero) independently with probability C/N, with 
C the mean variable connectivity, so that a prior for inclusion of an edge is 



^(A) = nf( 1 -S) S(A {ij) ) + ^-5(A {ij) 



(6) 

(ij) 

this being the connectivity in a standard Erdos-Renyi random graph. The couplings in 
the non-zero cases are described by a distribution with finite moments, and are sampled 
independently according to 

P(V) = J] p (v {ij) ) ; P(Vfo-) =x) = 4>{x) , (7) 

(ij) 

in the general case. A practical distribution for analysis is the ± J distribution defined as 

<j)(x) = (1 - p)S(x - J s ) + p5(x + J s ) , (8) 

with two parameters, p the probability that the link is anti-ferromagnetic, and J s the 
strength of coupling. Regular connectivity ensembles have each variable constrained to 
interact with exactly C neighbors, 

P(A)ocn^E^- C ) • ( 9 ) 

3. Representative parameterizations 

Four models are considered in greater detail owing to their simplicity and ability to make 
transparent a range of observed phenomena. The F-AF model includes ferromagnetic dense 
couplings (J = 0, Jo > (8)) and anti-ferromagnetic sparse couplings (p = 1 (8)), with 
connectivity C — 2, and is described by 

n(s) = -^p-J2 s ^ + j s J2 a m s ^ ■ ( 10 ) 

{ij) (ij) 



The function B(7, J s )/N is introduced to balance the ferromagnetic and anti-ferromagnetic 
tendency. Choosing 1(7, J s ) as a positive, monotonically increasing function of the scalar 
parameter 7 the relative strength of the anti-ferromagnetic and ferromagnetic parts are kept 
in some intuitive balance. As 7 increases there is an increased tendency towards aligning 
spins within the Hamiltonian - the ferromagnetic (ordered) state is promoted. 

It is also interesting to consider the converse case, the AF-F model with a ferromagnetic 
sparse part (p = 0) and anti-ferromagnetic dense model (J = 0,Jo < 0), with connectivity 
C = 2, 

H(S) = -J S A^SiSj + SiSj , (11) 

(ij) {ij) 

with B being again some suitably re-scaled function, J s must also be defined. 

These models can also be considered for the case of regular connectivity. Either anti- 
ferromagnetic couplings, the regular F-AF (10) and AF-F (11) models, are considered, but 
in each case with connectivity chosen to be C = 3 (a minimal choice above the percolation 
threshold). 



III. REPLICA METHOD 



The replica method is used in both [13, 15] to study the composite system free energy in 
the limit of large N. The replica method is the most concise analytical method available, 
although many results presented herein can be developed through the cavity method with 
suitable assumptions. For convenience the fields z ^> 0, as in (1), in the various calculation 
steps. Variations on this are useful in establishing a number of system properties as outlined 
in Appendix B. 

In the replica approach the typical case behavior is examined through the free energy 
density (3) averaged over the quenched disorder. That is to say we do not expect typical 
samples from the ensembles to differ in the value of the order parameters and other extensive 
properties. The replica identity 



(Z n )Q , (12) 

n=0 



2 dn 

allows for the average over the logarithm to be replace by the partition sum of a replicated 
set of variables. This is by an analytic continuation of n to the set of integers, giving a 
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form for which the quenched averages may be taken. The properties of the free energy are 
constructed through the replicated partition function 

n 

(z n ) Q = U 

a=l 

where the quenched averages and dynamic averages may be taken equivalently. 

The exponent is factorized with respect to the quenched variables in the sparse and dense 
parts. The average in the dense part involves an expansion to second order in N of J^-y 
The leading order terms are described by J and J 2 (4), and higher order terms are taken 
to be negligible in the large N limit. The average in the sparse part is more involved, the 
full method is presented in Appendix A. The brief outline of the method in the remainder 
of this section applies only for Poissonian connectivity in the sparse sub-structure. The site 
dependence in the energetic part is factorized in general by introducing three classes of order 
parameters 

= a7 E s " ; = ^ E s rsr ; *W = ^ E 6 s,s, ; ( 14 ) 

it i 

where q a describes the homogeneous magnetization, q{ ai , a2 ) describes the 2-replica correla- 
tions, and the generalized order parameter [16] <&(S) describes many kinds of spin correla- 
tions, where the bold font vector notation is used to represent a vector labeled by replica 
indices rather than site indices, denoted by an over-line vector notation, 

n 

6 S,S t = II 6 S",sr 5 S = {S a \a = l...n} . (15) 

a=l 

The order parameters q a and q{ ai ,a 2 ) can be defined from the generalized order parameter 
in the Poissonian connectivity case 

q a = *(*K ; 9<a 1)aa) = E H°)v ai v a2 ■ (is) 

(T a 

However, solving the saddle-point equations, by population dynamics in the RS descrip- 
tion, where order parameters are assumed to be invariant under replica-index permutations, 
is complicated without the redundant description (14), and the redundant description is 
necessary in the regular and F-F models. Furthermore having order parameters describing 
both dense and sparse parts is useful in discriminating effects due to sparse and dense sub- 
structures and the connection with the standard sparse and dense descriptions is also made 
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E^ 



. («■> 




(13) 



transparent in the limiting cases: taking q a = q( ai ,a 2 ) = to recover the thermodynamics of 
a sparse system; and $(<r) = 1 to recover a purely dense thermodynamic description. 

The original mixed topology problem is replaced by a site factorized (mean field) model 
- the complexity being encoded in a set of interactions amongst replica encoded in the order 
parameters. The definitions of the order parameters may be transformed to an exponential 
form by introducing a weighted integral over conjugate parameters (denoted by a hat). 
The exponential form allows a saddle-point method to be applied, an extremisation of the 
exponent allows the free energy to be identified as 

Pfe = S ^^M^ft.,^^,^.^} {W, *) + *) + *)} > (17) 

up to constant (ensemble parameter dependent) terms. The term Q\ encodes an energetic 
term describing interactions, which in the absence of an external field is given by 

- § logJ2 SiS ^(S)^(S') J dx<f>(x)e W {(3xj: a S a S^} , 

where <f>(x) is the coupling distribution in the sparse part (8). The term Q 2 is an entropic 
term coupling the sparse and dense order parameters 

S 2 = -logJ>xp i^ Q S Q + J2 q {ai ,« 2 )S ai S a2 +CHS) 1 . (19) 

S t a (ai,a 2 ) ) 

The coupling between the order parameters and their conjugate forms is present in the term 
g 3 = Cj2HSMS) + J2 ( laqa+ 9<«i,«2>?<«i,<w> ■ ( 2 °) 

S a (ai,Q!2> 

The free energy is used to calculate various self averaging properties of the system by tak- 
ing derivatives with respect to conjugate parameter, as outlined in Appendix B. The inverse 
temperature is conjugate to the energy, from which the entropy is calculated. Derivatives 

— * 

with respect to uniform fields conjugate to 1 can be used to test emergent ferromagnetic 
order. By inclusion of a random field of mean zero, the variance can be used to calculate 
correlation functions and susceptibility. 

The order parameters, defined at the extrema of the saddle-point (denoted *), obey 
coupled saddle-point equations 

$*(S) = V(S); q a * = J2S a V(S)- q* {aua2) = ^^^^(5) , (21) 

s s 
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where 

7» oc exp C^{a) + Y j q> a + £ T^^o** 1 , (22) 

^ a (0:1,02) J 

is a normalized probability distribution on the replicated state space. 

The conjugate parameters are determined by equations without coupling between the 
sparse and dense parts 



^(^oc^^^^expj^^rV^ ; 



Qa — P J 0Qa ; ?(oi,a 2 > — P J Q( ai ,a 2 ) 'i 

(23) 

with x distributed according to as in (7). From these six equations it is possible to 
eliminate the conjugate parameters (23) to leave a fixed point defined without the conjugate 
parameters. 

IV. REPLICA SYMMETRIC FORMULATION AND BELIEF PROPAGATION 
A. The RS saddle-point equations 

The order parameters are defined by the standard sparse and dense RS forms 

= / dh*(h) n ex 2 p C osTl } ; q «* = m ; > = q ; (24) 

with the variational aspects captured by the normalized distribution on the real line (77) and 
two scalar parameters (m,q). 

The saddle-point equations can then be written for the general case, inclusive of regular 
and Poissonian connectivity , as 

ix(h) cx I I f[ \dh c dx c Ti{h c )<p(x c )} S(h- h RS ) \ , (25) 

J \c=l / c e ,A 

where 

Ce 

h RS = m + Xy/q + ^ atanh (tanh(/3x c ) tanh(/i c )) , (26) 

c=l 

and c e is distributed according to the excess connectivity distribution, a normalized distri- 
bution proportional to CP{C — 1), where P{C) is the full variable connectivity distribution, 
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regular or Poissonian. The integration variable A is normally distributed. The dense parts 
are defined similarly 



m 



J (j[ [dh c dx c ir(h c )<f)(x c )} 5 (h - h RS ) tanh(/i) ^ , (27) 



^=1 / C/ ,A 

and c 

q= j /Y[[dh c dx c 7r(h c )(j)(x c )}5(h-h RS )t8,nh 2 (h)\ , (28) 

J \c=l / C/ ,A 

but with the averages in cj being with respect to the full connectivity distribution. 

These equations can be solved by a method of population dynamics [17] subject to two 
additional recursions on scalar quantities (27)- (28). 



B. Composite belief propagation equations 

Composite BP can be interpreted in the context of the composite system as a heuristic 
method of determining marginals of the static probability distribution (40), given a quenched 
sample [18]. Whereas an exhaustive calculation requires 0(2 N ) operations to construct a 
marginal, BP is guaranteed to produce an estimate in a number of operations that scales 
only linearly with the number of edges. 

The equations from factors to nodes are trivial in the case of binary factors, so iterations 
on variable messages alone can be composed. Defining two directed messages for every 
link (ij), which can be interpreted as log-posterior ratios for spins on graphs with some 
interactions removed (cavity graphs) 

h t? = ^Y. T ^ p{t+1) ^ = T ^ G ^ = \ E atanh(tanh(/341.)tanh(/3J w )) , 

' n ' k\{i,j} 

(29) 

where P is used to denote an approximated probability distribution. The cavity graph is 
a factor graph rooted in variable i with the coupling set to zero. The assumption 
underlying the probabilistic recursion is the independence of log-posterior ratios, which 
allows them to be used accurately as priors in each step, so that the recursion is equivalent 
to that on a tree. 

BP can be iterated from some initial condition. If correlations between messages are 
sufficiently weak then the messages will converge to correctly describe the probabilities. From 
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these marginal properties, such as the magnetization at equilibrium, can be constructed. A 
log-marginal may be estimated by 

H * t+1) = 4E rl0 S />(m) (^ = t\G) = i^atanh (tanh(/3^1 i ) tanh(/3 J {ij) )) . (30) 

T j\i 

The condition of sufficiently weak correlations is closely related to the notion of a pure 
state is statistical mechanics [2]. The assumption of independent messages applies only 
when the log-posteriors (29) reflect the distribution in a pure state, the similarity with (26) 
is not coincidental. Pure states act as local attractors of the BP dynamics, and it is only 
when there is a competition between these attractors that dynamics is expected to fail. 
With BP initialized sufficiently close (globally) to a pure state, or in the case of a unique 
attractor, convergence to the pure state can be anticipated giving a correct description of 
the equilibrium probability distribution. 



1. Simplification of dense messages 

Assuming the messages to be independent, then each message can be considered as a 
random object determined by the couplings in the cavity graph. The messages are therefore 
i.i.d. and the sum over many messages will converge to a Gaussian random variable. To 
leading order the messages may be rewritten incorporating this insight 

hfZf = m (t) + Vo^lilj + 4 Yl atanh ( tanh (/34tj tanh(/3J (ij) )) , (31) 

ke{di\j} 

where mf^ is the mean and the variance, and term di is used to denote variables connected 
to i through strong couplings. The distribution over reweighted messages Aj_>j will be 
asymptotically Gaussian if the approximation is correct. The value of the message for a 
particular instance of the quenched disorder is given by 

+ q {t) \fl j = - atanh (tanh(/341j tanh(/3J fo> )) . (32) 

^ k\{diUj} 

The Gaussian statistics are defined by analogy with the RS thermodynamic quantities, to 
leading order in 

N N 

mW^Jo^tanhpf) ; = (3 2 J 2 - £ tanh 2 ^) , (33) 

i=i i=i 
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FIG. 3: (color online) BP constructs an estimate of the posteriors by message passing, each message 
is a log-posterior estimate for some variable subject to the removal of the interaction with another 
variable (the variable to which the message is passed), as in the top sub-figures. In the lower two 
sub-figures the central limit is applied to the messages on dense links and in some cases only a 
single parameter is then required to represent the O(N) dense messages. A related approximation 
is implicit in the derivation of the RS free energy. 



for any dense set of couplings [19]. The log-posterior ratios for the spin states on the full 
graph are approximated as 

(3Hf +1) = m {t) + \/^Uf + atanh (tanM/^lj tanh(/3%))) . (34) 

ked t 

The term Aj is closely related to X^j, up to a correction of order 1/N, by removing the 
restriction on the sum in j from (32). 

In the case that J ^ it is necessary to evaluate A; for each link, still requiring 0(N 2 ) 
evaluations as in the original algorithm. To reduce computational complexity it may be 
valuable to marginalize over this if J <C Jo or if the sparse couplings dominate dynamics, 
but if J = it is sufficient to take = and algorithm complexity is reduced to O(N), as 
illustrated in figure 3. A method for combining messages in models comprising both densely 
and sparsely interacting components has been recently introduced [20]. 
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C. Stability analysis 



If the replica description correctly describes a single pure state, then this implies the 
spin glass susceptibility is not divergent in the thermodynamic limit. In the case of a 
sparse graph the tree-like approximation provides a natural basis for constructing a self- 
consistent estimate of the spin-glass susceptibility [21], whereas in the dense model a direct 
test of eigenvalue stability towards replica symmetry breaking can establish a complete 
description [22]. 

An analytic framework entirely within the replica method might be constructed to test 
spin-glass susceptibility. As in Appendix B, a connection can be made between the particular 
instability in the order parameter and the divergence of the physical quantity, spin glass 
stability, within the RS framework. This identity is not pursued within this paper, instead 
a more intuitive framework, believed to be equivalent, is presented. 

The non-divergence of the spin-glass susceptibility in sparse and dense models requires 
the local stability of the saddle-point equations; this proves to be an equivalent condition to 
the stability of the BP equations on a typical graph in the limit N — > oo [19, 21]. Stability of 
the BP equations is therefore explored for a typical sample. Assuming a linear perturbation 
{Shf^j} about some fixed point {hf^j} of the BP equations (29), implies an independent 
recursion on the perturbations that may be written at leading order 



In the dense part, the fluctuations may again be represented by a Gaussian random variable 
of mean and variance 



respectively, since the couplings are assumed to be uncorrelated with the perturbations in 
BP, the average is with respect to all perturbations and fields incident on j. An expansion of 
hi^j in terms of Hi is possible so that the statistics can be shown to be identical at leading 
order for all j [19], therefore the perturbations evolve according to quantities which are time 
but not site dependent 




i\{j,k} 



(1 - tanh 2 (/3/4^.)) tanh(/3% } ) 
' J 1 - tanh 2 ^/^,) tanh 2 (/5J fo) ) 



(35) 



Jo(^E,(l-tanh 2 (/3^.))) and ((5/^) 2 (l - tanh 2 (^)) 2 ) , (36) 




(37) 
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where 8H® are the perturbations in the log-posteriors, which are equal to Shf^j at leading 
order whenever is not a strong coupling term. 

A final approximation is to assume Hi is uncorrelated with SHi. In this case the statistics 
can be written only in terms of (5 Hi) and ((5 Hi) 2 ). However, this is not true at 
leading order when a sparse component is present. Variables with larger connectivity in the 
sparse part, are described by a field distribution of greater variance, and the perturbations 
scale similarly. Instead, the pair of correlation functions (37) determines the evolution of 
perturbations. 

Evolution of the perturbations can be undertaken in parallel with BP; to each message is 
attached a representative statistic for, or a distribution over, perturbations. It is sufficient to 
consider a distribution of perturbations characterized by a mean Shf^, and variance Sh 2 f}^j, 
attached to each macroscopic field. If these parameters decay exponentially, in expectation, 
then this is an indication of fixed point stability. 

Assuming that there is no linear instability, the equation determining Sh 2 ^^ is 

SVt? = ^ + E ( (1 ' ta " h2(/? f- ,))ta " h(/3J< - J>)N | \ (38) 

with a similar equation applicable to the case of a linear perturbation. 

The BP equations can be interpreted as a recursive instantiation of the RS saddle-point 
equations (25)- (28) except in the explicit site dependence, so that quenched disorder specific 
correlations may accumulate over several updates. Assuming a negligible feedback process 
in BP, or a modified problem without loops or with annealed disorder, the macroscopic 
properties established by BP will depend only on the steady state distribution of messages 
on sparse links and the mean and variance of dense messages. Objects analogous to a 
histogram estimate to ir (25), and scalar parameters and in the saddle-point method. 
However, at the level of the mapping of individual points in the RS description (26) it is 
possible that local fluctuations of the messages on fields are unstable, despite stability in 
the distribution. Whereas divergence in (5h) might be observed in a macroscopic instability 
in the first moment of n, an instability of the mapping in (5h 2 ^ will not be realized in any 
macroscopic moment of the distribution. It is this instability in the mapping which is probed 
by the BP stability analysis. In the absence of a linear instability it is assumed divergence 
in (5h 2 ^ is a necessary condition for any local instability. 
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The fluctuations on sparse messages are represented fully in this framework, whereas 
dense messages are summarized under approximation. The stability is a self-consistent 
(longitudinal) test of stability but is known not to probe all possible instabilities and so 
provides only a sufficient criteria for instability. The SK model is an example where the 
longitudinal stability of the ferromagnetic phase, as derived through a BP framework [19], 
does not capture correctly the spin glass transition at low temperature, as shown in figure 4. 
Since the models investigated in detail later have inhomogeneity in the sparse sub-structure 
only (J 2 = 0), it is felt the test of stability as applied in this paper may be a more accurate 
reflection of true local stability towards replica symmetry breaking. A connection between 
the stability tested through the BP framework, and an instability entirely within the replica 
method, might be established as outlined in Appendix B. 



V. EXACT HIGH TEMPERATURE FORMULATION 



In the limit [3 — > the paramagnetic solution $ = l,q a = 0,q{ai,a 2 ) = is the only 
stable solution, but becomes unstable as temperature is decreased. This process can be 
investigated by considering the moments of $ through a moment expansion representation 

$(<T) = l + ^2q a v a + J2 W^^ + E E «i • (39) 

a (ai,a 2 ) L=3 (a\,...,a L ) 

The saddle-point equations can be solved in each moment {q}, and stability tested in some 
subset of the moments. 

In the sparse sub-structure both the excess and full connectivity distributions are Pois- 
sonian, the saddle-point equation (21) can be expanded, using the identity (16), as 



Y[ [cosh(X i g (aii ... jQL) )(l + a ai ■ ■ ■ a aL tanh(X L g (aii ... jQi) )] 

(ai,...,a L ) 



pw = n 

L=\ 

eliminating the conjugate parameters (23). The terms 



(40) 



X 1 = (3 Jo + Ti ; X 2 = (3 2 J 2 + T 2 ; X t = T t if % > 2 , 



(41) 



determine transition properties where 



T i = C dx<p(x) tanh^x) . 



(42) 
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The saddle-point equations can be written in terms of an equation for each moment 

(1 - tanh 2 (X L g (ai ,..., QL) )) (g* . . . S°>) X L 

where the notation (• • -)^ x indicates an average with respect to (40), but with x = 0. A solu- 
tion is apparent which is the paramagnetic solution with z = (cr ai . . . a aL ) and q{ ai ,...,a L ) = 
for all choices of indices. This is the only solution when Xl — > 0, corresponding to the high 
temperature limit. 

At lower temperature a solution may emerge in one of the moments. It is only necessary 
to show that some component q allows a non-zero solution. The second term in (43) is 
zero at leading order in q in the moments of the distribution, and there is no coupling of 
the moments at leading order. Hence any solution which emerges continuously from the 
paramagnetic solution must do so with equality at leading order between the first term of 
the right hand side and the left hand side. This leads to a criteria Xl = 1 for the existence 
of a continuous transition. 

For a discontinuous transition to occur in some component, without > 1, requires the 
derivative of the second part with respect to q to be a convex function of q in some range 
of the parameter (43). However, the derivative is a concave function of q, so that unless 
Xi > 1 for some component, there can be no solution other than the paramagnetic one. 



A. High temperature phase transitions 

The existence of non-paramagnetic order is determined from (43) as: 
Xi > 1 1-spin / Ferromagnetic (F) order ; 

X 2 > 1 2-spin / Spin Glass (SG) order ; (44) 
Xl > 1 L-spin order . 

In each case the solution which emerges may be estimated by an expansion in the right 
hand side of (43) up to some order. Cubic order can be considered as a minimum to obtain 
the continuously emerging solution. To allow for the replica limit n->0an assumption on 
the correlations is required, RS being the simplest, and the order parameters may then be 
determined. Depending on the order of solution required, some coupling between moments 
is relevant, and it is necessary to solve a set of coupled equations. 
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The emergence of a ferromagnetic phase is realized in a continuous transition towards 
non-zero values of q a . Through coupling of the order parameters, all parameters q~{ ai ,...,a L ) 
become non-zero at order 0((q a ) L ). 

The emergence of a spin glass phase is realized in a continuous transition towards non- 
zero values of q{ ai ,a 2 )i while q a = 0. Even order parameters are generated by the coupling 
of terms at higher order. 

The transition towards an L-spin order is irrelevant to the high temperature analysis, 
since by consideration of (41) it is clear that X L < X 2 for all L > 2, with equality only 
in pathological cases, therefore the transition can only be towards a ferromagnetic or spin 
glass phase. 

In the case that Xi = X 2 , at the high temperature transition point both orders may 
emerge simultaneously and in competition. This case can be understood at leading order 
through an SK auxiliary model. 



B. SK auxiliary system 

In either the case of a ferromagnetic order, or spin glass order, the behavior is described 
at leading order about the paramagnetic phase by the terms {q a } and {q{ ai ,a 2 ) }• The 
free energy can be written in these cases as a function of only these two types of order 
parameter. After elimination of conjugate parameters the free energy can be written up to 
constant terms as 

\ S K a (ai,a 2 ) J ex (ai,a 2 ) 

(45) 

This is the replica formulation of the SK model free energy [7]. Therefore at leading order 
the high temperature phases are equivalent to the SK model, up to the (3 dependence of the 
energetic coupling terms. Instead of the standard term /3J there is X-y, and instead of (3 2 J 2 
there is X 2 . 

For every composite system of Poissonian connectivity there exists an auxiliary SK model 
with an equivalent leading order behavior at high temperature. By mapping the composite 
parameterizations to the SK model all the leading order high temperature transition prop- 
erties must carry over, including the nature of RSB and the stability of the RS description. 
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FIG. 4: (color online) The phase diagrams for disordered spin glass systems often exhibit a phase 
behavior similar to the SK model. Left figure: The phase transitions are indicated by solid dark 
lines. As temperature is lowered there is a transition from an RS paramagnetic phase (m = q = 0) 
to either an RS ferromagnetic (m > 0) or spin glass {q > 0,m = 0) phase. As temperature is 
lowered in the ferromagnetic phase there is also an RS to Full-RSB transition. Under the RS 
assumption the longitudinal instability measures calculated in the context of BP coincides with 
the F-SG transition in the RS description (dashed line). The instability of the ferromagnetic phase 
is not correctly predicted; the result is a lower bound in temperature for the replica instability in 
the ferromagnetic phase (towards a mixed phase). 

Let A denote the parameterizations (Jq, J a , (3 a ) of an SK model with an equivalent high 
temperature behavior to some composite system at the high temperature transition. This 
parameterization is redundant, there are only two independent parameters and so J A = 1 
is chosen. The standard phase diagram for an SK model under these parameterization is 
demonstrated in figure 4. 

The auxiliary parameterization is determined by the mapping equilibrating the coeffi- 
cients in the free energy (41) 



Where this mapping is continuous it is possible to consider how the auxiliary system param- 
eterization responds to variation of temperature (or some other parameter) in the composite 
system. Variation of (3 in the composite model is realized as a trajectory in the auxiliary 





(46) 
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model parameter space given by 

dJl = J -J S C (I -tanh 2 ([3 J s ) _ J_ 

d(3 A J 5 Ctanh(/5J 5 )(l-tanh 2 (/5J 5 )) P A ' 1 ' 

In the case that the couplings to higher order moments are small (X L <C 1 for L > 2), then 
the mapping may be applied with some confidence to lower temperature. Such a scenario 
will occur when the X\ and X 2 are dominated by the dense sub-structure terms, or when C 
is large in the sparse sub-structure. 



C. Beyond leading order 

The leading order approximation to the composite system differs from the SK model in 
the anomalous dependence of energetic components on [3. This observation alone is sufficient 
to account for many of the novel features of composite models reported at high temperature. 

About the ferromagnetic transition the term q a appears at leading order to provide a 
thermodynamic description. The magnitude of (q a ) 2 is proportional to A 1 — X 1 — 1 at 
leading order and at L th order the value is dependent on moments of the distribution up to 
(j(ai,...,a L )- The set of non-linear coupled equations can be solved in parallel at each order. 
The ferromagnetic phase is at leading order an RS phase so that an expansion with simple 
RS components will be stable at leading order. The full description of the ferromagnetic 
phase differs from the auxiliary system description at third or fourth order. 

The spin glass phase does not include any non-zero odd moments, and is described at 
leading order by A 2 = X 2 — 1, and at second order includes the term q( aita2t a z ,ai)- This term 
arises from the sparse sub-structure and so behavior deviates from the auxiliary model at 
second order. However, since even moments have positive coefficients, all with a monotonic 
dependence on (3, phenomenological properties may not differ significantly from the VB 
model which has been thoroughly studied (e.g. [10]). 

In the vicinity of the triple point, where both Ai and A 2 are positive, the terms q{ ai ,a 2 ,a 3 ) 
and q{ai,a2,a 3 ,a4) are relevant at second order. The literature developed in studying the VB 
model is sufficient to describe RS properties, and stability about the triple point [9, 23]. The 
leading order behavior gives a transition from an RS ferromagnet to a spin glass according 
to a balance in the components Ai = A 2 /2. The second order term in the sparse model 
indicates the existence of a mixed phase, with a refinement of the transition line. 
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The AT line is sufficient to describe stability of an RS solution in the dense model at 
all temperatures [22]. In order to correctly describe transitions in the sparse or composite 
models it is necessary to consider a wider range of eigenvalues [23], which cannot be evaluated 
other than numerically, except at the percolation threshold (absent in the composite model) 
or as a polynomial expansion truncated at some order. 

A stability analysis considering moments up to fourth order was recently presented [15]. It 
considers an RS description with inclusion of second order effects {q{ ai ,a 2 ,a 3 ), Q{ai,a2,a 3 ,a 4 )} 
0}, but with an analysis of instabilities restricted to variation in {q a , q( ai , a2 )}- This predicts 
a comparable splitting of the line Ai = A 2 /2 to those found for the VB model, but for some 
ranges of parameters a stable spin glass phase is incorrectly identified. Since only a restricted 
set of eigenvalues is considered this is not unreasonable, but demonstrates a weakness in the 
method. 

1. Regular connectivity 

The derivations of this section so far, beginning from (43) onwards have been specific 
to the case of Poissonian connectivity (19) and do not necessarily extend to composite 
systems with non-Poissonian connectivity. The replica theory is developed along similar 
lines to previous sections in Appendix A to be inclusive of the regular connectivity ensemble. 
The 1-spin and 2-spin dense sub-structure order parameters are determined by (14) and 
take zero values in the paramagnetic phase. The sparse sub-structure order parameter is 
different from (14) to be inclusive of non-Poissonian connectivity, but in general takes a 
value $ = 1 in the paramagnetic solution, and may be expanded as a set of moments (39). 
However, with the new definition q a ^ q a and q( ai ,a 2 ) 7^ Q(ai,a 2 ) i n general. Each of these 
order parameters corresponds to distinct physical quantities: q a (Q{a u a2)) are related to the 
mean magnetization (2-spin correlation), whereas q a ,q{ ai ,a 2 ) correspond to these quantities 
weighted by connectivity in the sparse sub-structure, as indicated in Appendix B. 

Along similar lines to the previous analysis, it is possible to consider the emergence of 
order by treatment only of the leading order behavior about the paramagnetic solution. 
The 1-spin order terms are coupled at leading order by the saddle-point equations, thus 
there is no decoupled representation describing emergence of spin glass and ferromagnetic 
order in general. The criteria for a ferromagnetic solution to emerge continuously from the 
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V 9a ) 





(48) 



paramagnetic solution as temperature is lowered is determined by the point at which 

/3 Jo Ti tanh(/3x) 
c 

if such a point exists; its existence requires the principal eigenvector of the matrix to be 
the one vector. However, the existence of a solution point in the coupled equations is not 
guaranteed, and there exist a range of parameters in which decreasing temperature results 
in a pair of complex conjugate eigenvalues which exceed one in modulus. 

The right hand side of (48) represents the leading order 1-spin terms in the saddle-point 
equations (21), after elimination of the conjugate parameters. In the case of Poissonian 
connectivity the existence of a continuous transition is necessary for the existence of a 
ferromagnetic or spin glass phase (43). This is due to the concavity of the saddle-point 
equation when interpreted as a mapping, concavity is assumed to hold also for the regular 
connectivity composite system. 

However, in a general model it is necessary only for the principal eigenvalue of the ma- 
trix (48) to exceed one for some non-paramagnetic solution of the saddle-point equations to 
exist. When the modulus of the principal eigenvalue exceeds one the assumption of weak 
coupling between the moments in the order parameters expansion ceases to be valid when 
considering an expansion coincident with the eigenvector. The criteria that the modulus in 
the leading order expansion is greater than one corresponds to a set of criteria 



§JqT\ 

c 



> 1 1-spin order 



(PJ + %±Ti) ± y / ( / 3J + % i T 1 ) 2 + 4 
(/3 2 J 2 + %±T 2 ) ± V / (/? 2 J2 + %lT 2 ) 2 + 4^k > i 2-spin order ; (49) 



^r-T L > 1 L-spin order . 

The potential exists for the modulus to exceed one whilst the discriminant is less than zero 
in the 1-spin order term, when either 7\ or (3J are negative. This phenomena, absent in the 
VB and SK models, is contingent on a sub-set of couplings being anti-ferromagnetic. In spite 
of a comparable functional form in the 2-spin order term, the transition from a paramagnet 
to a spin glass is always described by a non-negative discriminant, and real eigenvalues. 

The complex eigenvalues imply complex conjugate eigenvectors. Where the eigenvalues 
are real it is possible to test the stability of the equilibrium solution by inclusion of a 
conjugate field in proportion to the eigenvector components (see Appendix B). However, 
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where the eigenvalue is complex such a field is not physical and is not consistent with 
assumptions made in the development of the equilibrium solution. 

Attention is restricted to real valued perturbations of the order parameters, which can 
be associated with the real valued conjugate fields. A local instability in the paramagnetic 
solution is only anticipated towards a ferromagnetic phase when the real part of the principal 
eigenvalue is larger than one, or towards a spin-glass solution when criteria (49) is met. 
If the paramagnetic solution is stable with respect to an infinitesimal term conjugate to 
the magnetization in the Hamiltonian then the paramagnetic solution will be recovered 
continuously as the conjugate field approaches zero. This is equivalent to the criteria that 
the linearized saddle-point equations are convergent to the zero solution. Linear instability is 
apparent when the real-part of the eigenvalues exceed one. However, since the perturbation 
is not coincident with an eigenvector there is no leading order solution to the linearized 
equations when the external field is added. The instability in the paramagnetic solution is 
towards a discontinuously emerging solution. 

The discontinuously emerging solution from the paramagnetic instability might be a lo- 
cally stable (thermodynamic or metastable) solution across a wider range of temperatures 
than that indicated by the local stability analysis of the paramagnetic solution. In limited 
simulations, comparable in size to those described in section VIII, the behavior observed 
at temperatures close to (but below) the modulus one criteria (49) is consistent with the 
hypothesis of two locally stable solutions. One solution describes the thermodynamic phase, 
and the other a metastable solution, with decreasing temperature a discontinuous thermo- 
dynamic is anticipated. 

The case of large 7 allows only for a transition from a paramagnetic to ferromagnetic 
state, and this may be discontinuous. As well as a thermodynamic solution, several dynam- 
ical transitions may describe changes in local stability criteria of the solutions; these local 
instabilities may dominant aspects of dynamics, and in general will not be coincident with 
thermodynamic transitions. 

At intermediate 7 values the paramagnetic solution may be locally unstable first towards 
a spin glass solution as temperature is lowered. The presence of another metastable or ther- 
modynamic ferromagnetic phase may change the properties of this transition by comparison 
with the standard continuous case. 

In the limit of large C, a simplified description is possible in the transition criteria for 



24 



the regular connectivity case. With a sensible scaling of the moments of <f>(x) so that Ti and 
T 2 remain finite as C increases, the final term in the discriminant (49) becomes negligible 
and a simple transition criteria is recovered, consistent with the Poissonian system 

PJ + T 1 >1; p 2 J 2 + T 2 >l. (50) 

This is also the result that would be obtained in naively applying the dense system method, 
using only a mean and variance of link strengths, to the two scale system. Examples of dis- 
continuous high temperature transitions are examined in section VI C, with a clear departure 
from the conditions laid out in (50). 

VI. LEADING ORDER PREDICTIONS FOR PHASE BEHAVIOR 
A. The F-AF model 

The SK auxiliary model can be used to predict trends as temperature or some other 
parameter is varied in the F-AF model about the high temperature transition points. Using 
the mapping (46) combined with an exact (FRSB) description of the transitions and phases 
of the SK model at high and low temperature, the trajectories implied by the mapping can 
be used leading order predictor of phase behavior. 

Choosing the F-AF models (10) such that 

B = 7 ; J s = atanh(l/v / C) ; (51) 

a class of models parameterized by 7 G [0, 00) is created. The disorder in couplings decreases 
with 7 from a typical spin glass set to an ordered ferromagnetic set. These models are 
characterized by a high temperature spin glass transition at (3c — 1 when 7 < 1, and a high 
temperature ferromagnetic transition at a temperature (3^ 1 = 7 when 7 > 1. There is a 
triple point in the parameter space at 7 = 1, (3 — 1. Phase transitions between ferromagnetic 
and spin glass phases are possible where (3 > 1 and 7 ~ 1. 

Near the triple point model parametrization (7 = 1) a decrease in temperature results 
in a competition between ferromagnetic and spin glass solutions. A graphical answer to 
which solution dominates is provided by figure 5, for a range of high temperature transition 
properties. If only leading order moments are considered in the free energy then all composite 
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FIG. 5: (color online) The F-AF models (10) in a parameter range (7 = [0.75, 1.25], 1//3 = (0, 2]) are 
mapped through (46) to auxiliary SK models parameterized by (J A / 'J A ,1/ ' [3 A ). These models are 
equivalent about the high temperature transition lines, and elsewhere equivalent when constraining 
higher than second order moments to zero (39). Horizontal isobars indicate constant (3, and the near 
vertical isobars indicate constant 7, in the composite model parameter space. The set of transition 
lines for the SK model are shown, the upper most solid lines describing the high temperature 
phase transition. The SK auxiliary model predicts that as temperature is lowered in the composite 
models behavior converges towards a mean field ferromagnetic behavior. For small 7 the prediction 
is that a spin glass phase transforms through a mixed phase to an RS ferromagnet behavior as 
temperature is lowered. Decreasing temperature about the triple point (7 = 1) there is only an 
RS ferromagnetic behavior. The three highlighted isobars correspond to composite systems from 
left to right parameterized by 7 = 0.952 (j£ = 0.925 at (3 C ), 7 = 1 (Jq = 1 at G ) and 7 = 1.23 
(Jq = 1.15 at (3c), across a range of temperatures. 

systems evolve towards an RS ferromagnetic behavior with decreasing temperature. Thus 
unusual transitions away from FRSB spin-glass phases towards RS ferromagnetic phases are 
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predicted as temperature is lowered. 

The auxiliary model is an approximation except very close to the high temperature tran- 
sition, where higher order moments are negligible and linear approximations apply. At lower 
temperature complicated couplings with these higher order moments may prevent an FRSB 
to RS transition. However, working at the level of linearisation near the transition point the 
unusual FRSB to RS transitions are still observed in some models. 

In the F-AF model a spin glass phase with zero magnetization can not be a sufficient 
description at low temperature. This is because the spins disconnected from the sparse sub- 
structure can evolve independently and undergo an independent phase transition induced by 
the dense sub-structure. The results at leading order are in agreement with this observation. 

B. The AF-F model 

Consider the assignments 

B = 7(1 - Ctanh(Jg/ 7 )) , J s = atanh(l/v / C) , (52) 

as applied to the AF-F model (11), with 7 e [0, Jg/atanh (1/C)] describing the level of 
order in couplings. Larger 7 can be considered, but these correspond to systems with small 
ferromagnetic couplings in the dense part rather than anti-ferromagnetic ones. 

The predictions based on a leading order representation of the order parameters are 
shown in figure 6. Composite systems are predicted to evolve towards spin glass phases 
as temperature is lowered; lowering temperature at large 7 results first in transitions to a 
stable RS ferromagnetic phases then towards a mixed phase before finally a spin glass phase. 
The auxiliary model predicts that at lower temperature the magnetic moment is suppressed, 
for all 7 up to the maximum value Jg/atanh (1/C), so that in the low temperature limit 
all systems are in a phase equivalent to a "finite temperature" spin-glass phase in the SK 
model. As temperature is lowered RS states become unstable towards RSB, which is the 
scenario normally observed in dense or sparse spin glass models. 

The prediction that all systems converge towards a finite temperature spin glass is a 
consequence of the limited moment description. The spin glass behavior is a residual effect 
of the sparse couplings, and at low temperature depends strongly on higher order moments 
which are absent in the auxiliary model. The spin glass phase is not induced by the dense 



27 



1/p 



A 



1.4 



1.2 



1.0 



0.8 



0.6 



AF-F 



SG 



l/p->0 



0.0 



0.2 



0.4 



1/(3=2 



J =0.925 




FIG. 6: (color online) . The AF-F model (11) as parameterized in 7 — [3 space (7 = 
[0.75, 1.25], 1/(3 = (0,2]) is mapped (46) to an auxiliary dense model parameter space. The aux- 
iliary model prediction is that the magnetic order parameter (m 2 ) goes to zero in all composite 
models as temperature is lowered, a FRSB spin glass phase describes the zero temperature limit. 
The three highlighted systems correspond to systems with 7 = 0.746 {Jq = 0.925 at (3c), 7 = 1 
(j£ = 1 at (3 C ) and 7 = 1.23 (J^ = 1.15 at (3 C )- 



anti-ferromagnetic couplings. 



C. Regular connectivity models 

Figure 7 demonstrates the limitations on the parameter range consistent with unique 
locally stable RS solutions, in the case of regular connectivity systems. The two figures 
correspond to the models (10) and (11), but with regular couplings (9) and mean connectivity 
3. The coupling scaling is 

B = 7(1 — Ctanh(J s / 7 )) ; J s = atanh(l/VC - 1) . (53) 
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FIG. 7: (color online) This figure demonstrates the critical points in the linear analysis of the 
paramagnetic solution for regular connectivity models. In this figure the horizontal line indicates 
a high temperature instability in the paramagnetic solution towards two spin order. Other lines 
indicate instabilities towards 1-spin order: the straight diagonal line is assuming the conditions (50), 
the thick and thin lines are the points where the real part or modulus of the principal eigenvector (s) 
equal one, respectively. Left figure: In the AF-F model decreasing temperature results in either 
a continuous spin glass or ferromagnetic transition, (a) At small 7 a spin glass phase emerges 
continuously with decreasing temperature, (b) At large 7 eigenvectors describing 1-spin order are 
real, a continuous ferromagnetic transition is found. Right figure: In the F-AF model continuous 
and discontinuous transitions occur, no continuous transition triple-point exists, (a) At small 7 
eigenvectors describing the 1-spin order are complex, but a spin glass high temperature transition 
is dominant, (b) At large 7 a continuous transition occurs described by a real eigenvector, (c) 
An instability in the paramagnetic solution in the first moment is anticipated at the lower (thick) 
line for intermediate 7, the properties of the discontinuously emerging solution (labeled ?) cannot 
be established by a linearized approach. The thin line indicates instability in the modulus for the 
linearized system, which is speculated to relate to the existence of the non-paramagnetic solution. 



29 



The choice of J s ensures that everywhere temperature (5 — 1 corresponds to a spin glass 
instability in the paramagnetic solution. The choice of scaling means that under the ap- 
proximated ferromagnetic transition scheme (50), the critical temperature implying local 
instability in the paramagnetic solution towards ferro magnet ism increases linearly with 7, 
denoted by the dashed line in figure 7. If the transition were predicted by (50) then a triple 
point would occur at 1: for 7 < 1 all high temperature transitions would be of a spin glass 
type; and for 7 > 1 transitions would be of a ferromagnetic type. 

With couplings characterized by (53), a range of 7 values allow the instability of the 
paramagnetic solution towards one-spin order to be described by complex eigenvalues. In 
the AF-F regular model, at small values of 7, the complex eigenvectors describe the stability 
of the paramagnetic solution towards 1-spin order, but as temperature is lowered a spin glass 
instability is first attained. At larger 7 (equivalently Jo) a triple point is reached, but here the 
eigenvectors are real, and a continuous transition from the paramagnetic to ferromagnetic 
phase is found at larger 7. 

In the F-AF regular model complex eigenvalues occur in a parameter range relevant to 
the high temperature transition. When Jo is sufficiently large a continuous high temper- 
ature ferromagnetic transition is observed, and at small 7 there is a continuous spin-glass 
transition. There exists a broad range of 7 between these regimes where the ferromagnetic 
solution can not emerge continuously from the paramagnetic solution and two locally stable 
solutions are anticipated. There is no triple-point in this model suitable for a perturbative 
analysis. 

In a small number of Metropolis-Hasting Monte-Carlo simulations [24] two attractors 
corresponding to paramagnetic and ferromagnetic type configurations were found in these 
parameter ranges, though no systematic analysis was undertaken. 

VII. REPLICA SYMMETRIC SOLUTION OF LOW TEMPERATURE BEHAV- 
IOR 

In figures 8,9 and 10 stability measures and magnetizations for the composite models, 
equivalent at f3c to SK models with J$ =1, J5 4 = 1.15 and = 0.925, are presented at 
various temperatures below the l/(3c- The trends found are compared to those predicted 
by the auxiliary model in the vicinity of the transitions, as shown in figures 5 and 6, and 



30 



also RS solutions to dense (SK) and sparse (VB) models with equivalent high temperature 
properties. 

A. Numerical evaluation of the saddle-point equations 

To work beyond a perturbative approach the RS saddle-point equations are solved by 
population dynamics [17]. The results are presented based on samples from a single run of 
a population dynamics algorithm. In population dynamics machine numbers are used for 
m and q and the distribution 7r is represented by an order-parameter histogram (W) of N 
components 

n^W = {h 1 ,...,h N } . (54) 

The saddle-point equations (25)- (28) are treated as a mapping with integrals and summa- 
tions replaced by random samples. This implies a random map from the histogram to itself. 
Updating Histograms recursively by a large number of random maps, from a random initial 
condition, leads to an accurate description of the fixed point n. 

The random sampling is done in such a way as to reduce fluctuations in the variance 
of the Gaussian distributed samples, and mean of the Poissonian distributed samples, to 
0(1/N). A single iteration includes an update of every field in the histogram W with either 
parallel or random sequential order. Given that anti-ferromagnetic couplings play a role 
in the dynamics, there is a risk that an invalid macroscopic anti-ferromagnetic state could 
be amplified by parallel updates. This scenario does not form a problematic point in the 
analysis undertaken, but was relevant to work undertaken in [15], and carefully avoided. In 
order to control finite size effects a scheme of microcanonical sampling was employed with 
respect to W, so that each field in generation (t) is involved in forming exactly C fields in 
generation (t + 1). 

A histogram of 65556 floating point fields run for 1024 iterations appears to resolve all 
statistical quantities of interest down to a temperature of ~ 1/(10/^), with great precision, 
even in the vicinity of phase transitions. At lower temperature there is a rapid decrease in 
the resolution of statistical quantities, which is uniform across tested systems and probably 
related to numerical precision limitations in the representation for hyperbolic functions. 
Based on the converged set of order parameters samples are taken in the following 256 
iterations to determine robust system statistics. 
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The initial condition for the order parameters m 2 , q and W are chosen as paramagnetic, 
combined with a small systematic bias towards spin-glass and ferromagnetic configurations 
with small, but non-zero values to the dense sub-structure moments (m 2 = q), elements of 
W are sampled according to a Gaussian J\f(m, q) such that the mean and variance of the 
histogram values are m + 0(l/N) and q = 0(1/N). Other initial conditions were also tested 
to ensure that dynamical bias was not implied by initial conditions, the suggested scheme 
converged effectively and systematically. 

1. Numerical evaluation of the stability equations 

The longitudinal stability is tested by initialising a fluctuation histogram SW 

^ = {(x 2 )f ) ,(x 2 )?,...,(x 2 )S ) }, (55) 

where each component corresponds to a distinct field in the histogram W (54). Each com- 
ponent represents a topology free measure of 5h 2 ^j, each of which is evolved according to 
(38), with the site dependent fields and parameters replaced by a sample of fields from W 
and other quenched disorder determined as in the field update. Cases in which J 2 = 
= o), without linear perturbations are considered. The stability exponent is 

A« = log E ' (x2 ( ) f 1) , (56) 

which is negative if BP is convergent in expectation. This is averaged over many generations, 
alongside renormalization of 5W to prevent numerical precision problems. 



B. The F-AF and AF-F models 



Results for VB, SK, F-AF and AF-F models are shown. The VB model presented for 
comparison is of connectivity 2, the same as the sparse sub-structures for F-AF and AF-F 
models, and has a balance of anti-ferromagnetic and ferromagnetic interactions described by 
a ±J model (8). Figures 8, 9 and 10 follow the dashed lines in figures 5 and 6 to highlight 
the behavior of the system as temperature is lowered along fixed Jq values. Figure 8 
demonstrates the results for the set of systems equivalent at the high temperature transition 
point to a dense model with Jq = 0.925. In all systems there is a high temperature transition 
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FIG. 8: (color online) A comparison of the stability exponent and magnetization for the F-AF 
(circles), AF-F (crosses), VB (dashed line) and SK (solid line) models under the RS assumption. 
Every model is equivalent at the high temperature spin glass transition point to an SK model 
parameterized by Jq = 0.925, and temperature variation is considered on the rescaled interval 
ftc/P = [0.2, 1.05]. In the top figure two stability exponents are given for the SK model, a longi- 
tudinal measure SK(RS) and a latitudinal measure SK(RSB). In the lower figure the sparse and 
dense models show similar trends with A > and m 2 = 0. Composite models behave as sparse 
spin-glass models whenever m 2 = 0, but there is a departure in both models at low temperature. 
In all models as temperature decreases A > 0, except for the F-AF model which is negative over an 
intermediate temperature range. Both composite models attain a non-zero magnetic moment at 
low temperature, which is not seen in the VB or SK models. The F-AF model is in approximate 
agreement with figure 5 at high temperature. However, the behaviors observed in the composite 
models at low temperature are not anticipated by the auxiliary model. 



that is P — SG at 0c — 1, behavior is examined for relative temperature fie /ft in the interval 
(0.1,1.05). 

The stability exponent (A) and magnetization (m 2 ) are identical in all the models very 
close to the transition, the phase is a spin glass (m = 0,q > 0) and the RS description 
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is unstable (A > 0). The F-AF model becomes unstable towards a mixed (unstable RS 
ferromagnetic) phase at relatively high temperature. This is qualitatively similar to the 
prediction based on the auxiliary model of the composite system (see figure (5), and the 
transition temperature is comparable to what would be predicted by the auxiliary model. 

When the magnetization is zero (the spin glass solution) only the even moments of the 
distribution in the composite models contribute to their behavior. These include only sparse 
model dependent parts for F-AF, AF-F so that these models are described by a saddle-point 
solution identical to the sparse model. 

In the AF-F model the ferromagnetic order parameter is suppressed down to a tempera- 
ture fid ' P ~ 0.25 where it acquires a small value. This is close to the point where q reaches 
a maximum value, saturation is reached before q — 1 due to the disconnected component in 
the sparse sub-structure. This low temperature transition must have a strong dependence 
on higher order moments since it is in strong contrast with the auxiliary model prediction 
(figure 6). 

Figure 9 demonstrates results for the same models and temperature range, but for cases 
in which the models have a high temperature triple-point transition. In this figure the F-AF 
model has a behavior that is clearly distinct from the other three models. As temperature 
is lowered a ferromagnet phase is found rather than a spin glass phase in the other cases, 
in agreement with figure 5. At lower temperatures a maximum magnetization is reached 
and a small decrease in magnetization is discernable at the lowest values in the temperature 
range. With Pc/P < 0.5 the RS ferromagnetic phase becomes unstable to a mixed phase. 

Initially, at high temperatures, the AF-F model is described by a spin glass phase. With 
the continuous emergence of a ferromagnetic moment at low temperature there is a decrease 
in the stability exponent. 

In figure 10 the behavior of systems exhibiting a high temperature ferromagnetic transi- 
tion are shown, systems with auxiliary models defined by Jq = 1.15 at the high temperature 
transition. In this regime reentrant behavior is seen in the SK model, but not in the VB 
or composite models. The two composite models follow very closely the behavior of the 
VB model, although at Pc/P ~ 0.3 there appears to be a modification of the trend in the 
stability exponent for the AF-F model absent in the F-AF and VB models. 

The ferromagnetic moment is largest in the AF-F model at high temperature, and the 
F-AF model at low temperatures. There are also several such cross overs in the stability 



34 



-0.1 



0.2 



0.3 



0.4 



0.5 



0.6 



0.7 



0.8 



0.9 



P c /P 



0.1 




SK(RSB) 

SK(RS) 



CM 



E 0.0! 



Ql IE 

0.2 0.3 



P c /P 



FIG. 9: (color online) A comparison of the longitudinal stability and magnetization for the F-AF 
(circles), AF-F (crosses), VB (dashed line) and SK (solid line) models under the RS assumption. 
Every model is equivalent at the high temperature transition to an SK model with Jq = 1, 
coincident with the triple point in the phase diagram. Temperature variation is considered on the 
rescaled interval (3c/P = [0-2, 1.05]. Trends differ in F-AF from figure 5 in that the magnetization 
acquires a maximum value, and the stability exponent tends towards a positive value at sufficiently 
low temperatures. Trends differ in AF-F from figure 6 in the appearance of a magnetic moment at 
low temperatures. 

exponent. The RS solutions are stable for the composite systems and VB over the full 
temperature range presented. 

VIII. REENTRANT BEHAVIOR AND STRUCTURE IN FINITE SYSTEMS 
A. BP and Monte-Carlo simulation 

Some testing of thermodynamic results was undertaken in samples of iV = O(100)-O(8000) 
spins by sampling through a Metropolis-Hastings algorithm [24], and estimating log-posterior 
ratios by BP. These studies verified qualitatively the outcomes of the thermodynamic anai- 
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FIG. 10: (color online) A comparison of the longitudinal stability and magnetization for the F-AF 
(circles), AF-F (crosses), VB (dashed line) and SK (solid line) models under the RS assumption. 
Every model is equivalent at the high temperature ferromagnetic transition point to an SK model 
with Jq = 1.15, and temperature variation is considered on the rescaled interval (3c / P = [0.2, 1.05]. 
Two stability exponents are given for SK. The marginal stability at the paramagnetic-ferromagnetic 
transition point (ftc/ft = 1) is with respect to a linear instability, which is captured by the longitu- 
dinal instability exponent [SK(RS)], but not by the other non-linear stability exponents. Properties 
of the F-AF model display features of the VB model rather than the auxiliary model predictions 
(figure 5). Trends also differ in AF-F model from figure 6, as instability is not realized until much 
lower than the predicted temperature, properties are again closer to the VB model. 

ysis at high temperature. The paramagnetic phase was observed to transform continuously 
into either a ferromagnetic, spin glass, or mixed (unstable ferromagnetic) phase as temper- 
ature was decreased. The ferromagnetic state is assumed to be described by a connected 
phase space up to finite size effects. Stability of the BP algorithm was measured through 
the mean square change in BP log-posterior estimates (30) 

1 N 9 

i=i 
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Number of BP message updates Variable Connectivity 

FIG. 11: (color online) Results in applying BP and Monte-Carlo simulation to an F-AF model of 
size 5000 spins, and 7 = 1 for various temperatures. Left figure: Iteration of BP on a sample graph 
from various initial conditions is convergent for this sample of quenched disorder at intermediate 
temperature only, as indicated by the exponential decay in the stability measure. Right: For the 
case where BP converges (3 = 1.5, the mean and variance in the field distribution are demonstrated 
as a function of variable connectivity in the sparse sub-structure. Thick lines (circles) demonstrate 
the results of Metropolis-Hastings Monte-Carlo simulation. Thin lines (crosses) demonstrate the 
estimates of BP. These are in agreement except at high variable connectivity. The magnetism 
of the system is supported by the alignment of low connectivity variables, with variables of high 
connectivity in the sparse sub-structure being magnetized in the opposite direction. 

this being a new definition of A related to (56), but distinguished by the algorithmic context. 

Figure 11 demonstrates a simulation of an F-AF model with 5000 spins. This demon- 
strates that the non-monotonic behavior seen in the RS solution of the F-AF model, and 
predicted by the leading order expansion, can be realized in finite systems also. The second 
part of the figure demonstrates the structure of the magnetic phase in the F-AF model. The 
macroscopic magnetization is supported primarily by spins coincident with the disconnected 
component in the sparse sub-structure. 
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B. Structure of phases and transitions 

In the F-AF model the inhomogeneity in magnetizations, with the disconnected com- 
ponent being the most strongly aligned set of variables, seems an intuitive and necessary 
feature in a model with such a stark contrast in coupling types. 

The disconnected component appears to play an even more vital role in the AF-F model. 
In the magnetic phase of this model all the disconnected components are observed in Monte- 
Carlo and BP experiments to be anti-correlated with the macroscopic magnetization, which 
is an intuitive result. Whereas almost all other variables, connected through the sparse 
sub-structure take values aligned with the macroscopic order. In the large system limit 
there should be some discrimination in the topology within the sparse- substructure. Some 
important topological features of sparse Poissonian graphs are outlined in figure 1. In general 
the highly connected spins may take one alignment, the disconnected component an opposite 
alignment, with other variables intermediate. 

The inhomogeneity in the structure must also be vital in allowing continuous transitions 
between various phases, and in the dynamics of models. The continuous emergence of a 
magnetic phase as temperature is lowered in the AF-F model is presumably by a nucle- 
ation process, whereas in the F-AF model the ferromagnetic part can emerge first in the 
disconnected component and percolate inwards to the core of the sparse sub-structure. The 
absence of sufficient inhomogeneity in the regular connectivity models is responsible for the 
metastability found in some parameter ranges. 

IX. SUMMARY 

We have investigated composite models that combine well-studied disordered spin systems 
with densely and sparsely interacting components with the expectation that the combined 
model would exhibit new feature that have not been observed in the original systems. Here 
we focused on the case of competing interactions of ferromagnetic and anti-ferromagnetic 
nature and have explored the phase diagrams for different interaction types and relative 
strengths. The equilibrium properties were examined through the replica method, exhibiting 
a new reentrant behavior from spin glass to ferromagnetic phases as temperature is lowered, 
and transitions from replica symmetry broken to replica symmetric phases. A discontinuous 
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transition from the paramagnetic to ferromagnetic phases has been observed in regular 
connectivity graphs. 

While the models investigated here comprise classical and well-understood models, they 
provide insight and new understanding of simple complex systems that combine different 
structures, and different levels and interaction types. Such systems have been increasingly 
the subject of interdisciplinary research activities using methods of varying mathematical 
rigor. We believe that established methods of statistical physics are highly suitable for 
understanding both macroscopic and microscopic properties of such systems and that the 
current investigation paves the way for the study of similar complex systems. 
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APPENDIX A: REPLICA CALCULATION 

Carrying out the calculation via the replica method involves a combination of sparse 
and dense quenched disorder averages. It is convenient to define the sparse substructure in 
terms of an adjacency matrix A: labeling each edge by /i and each variable k, = {0, 1}. 
With mean connectivity C the number of edges is CN/2 so that in the absence of other 
constraints, the probability distribution is denned 



This is a micro-canonical description of interactions, but formulations with the number of 
edges not strictly fixed (to CN/2) are possible. In the limit of large N this describes a Pois- 
sonian distribution in the variable connectivity. Both Poissonian and regular connectivity 




(Al) 
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are sufficiently described in typical case analysis by 



CN/2 



p(a) oc n 



AT 



n(^(£4*- c ')) n p (^), (A2) 



i=i 



the average in c/ being with respect to the marginal variable connectivity distribution of 
mean C [25] and taking P(A fl k) to be a sparse prior 



The Hamiltonian may be written in a form 

2 



2 \ 2 



(A3) 



(A4) 



where the representation of the dense part is unmodified from (1), is the random sparse 
coupling sampled according to <p(x) in (7), but can be replaced by the integration variable 
x in the self averaged expressions. The replicated partition function is 



(A5) 



(Z n ) Q = njE5«](n,(exp{fxE Q [(E fc 4*3?) 2 -2]}) a 

X n fo) (exp{^g,E Q ^}) jI5 • 



Since the Hamiltonian is factorized with respect to the sparse and dense quenched variables, 
these averages may be taken independently 

In the sparse part it is useful to linearize the squared components with a Hubbard- 
Stratonovich transform for each factor node and replica index pair 



>a = / n [ DA a (n ( ex p n 



expi V^E A ^ 



xl A 



with 



DA = 



d\ exp <* — — 
2tt ./ 1 \ 2 



(A6) 



(A7) 



The delta functions in the adjacency matrix probability distribution (A2) can be repre- 
sented by complex contour integrals 



V k ) fi J k \ }x J k fi 



J k > 
(A8) 
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using the notation 



f D - x -^if§-- /" v II/" 



(A9) 



The contours can be taken around the origin in the complex plane, so the identities are 
assured by Cauchy's residue formula. 

Factorizing various terms (A6) becomes 



(•••) A oc lj>T> e Z n/exp{-/*m} J V\j>T> 2 Y»\[ 



(A10) 



x C 



where DA = n^a-^M' G abbreviates a number of terms factorized in k. Subject to three 
integrals and an average on connectivities the dependence on the adjacency matrix is fac- 
torized in \i and k, and the trace over A can be completed 



n 



n 



(l - |) + ^Z k exp | K S % 



(AH) 



Taking the integral in picks out only the residue, the second term in the expansion, so 
that 



n 



£ f(-V)G 



«e n 

(*i> i={»j'} 



Introducing an identity 



i - En^^ ' 

CT a 

and extracting the k dependence through the order parameter definition 
1 = 



(A12) 



(A13) 



(A14) 



all quenched variable dependence is factorized except in the order parameter definition. In 
principle the identity applies to the entire complex plane, but relevant part of the order 
parameter is assumed to be real, the imaginary part takes a value in the final saddle-point 
formulation by assumption. The results are self-consistent given this assumption, and the 
necessity of real valued order parameters can be demonstrated in some special cases (see 
Appendix B). 
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Having taken the average in A the Hubbard-Stratonovich transform may be inverted to 
give up to ensemble dependent constants 

(•••>a = fUadH<T)U k (M ^c k Z k Ua^ {H^) - ^E k Z k S Skj(T )) Ck 
x n, [Er,«r HtM*) (exp {(3x £ a rV}> J 

where the average is with respect to the sparse coupling distribution (7). 

The dense part of the Hamiltonian can be expanded to second order allowing averages in 
the quenched couplings 



(• • •>,» = n 

(ij) 



1+ ^E^+i^ E srsrsfsf 

a (ai,a 2 > 



(A16) 



Defining the dense order parameters 



(A18) 



= jf E S * 5 9<«i.«> = a7 E ^f 2 , (A17) 
the dependence is extracted 

(• • •>,» = n a s (?« - £ e, n <QllQ2) ^ ( w 2 > - £ £* ^r) 

x exp {^EM exp E W , a2> 4^ 2 >} 

The definitions of q a , q{ ai ,a 2 ) an d introduced as 5-functions may be Fourier transformed 
introducing conjugate parameters, for the real part of $(<x) 

^(^-Izfc^fo,^ = jf exp{-CiV<l>( CT )<l>}exp|-C^Z fc <l>( CTfc )| (A19) 

By either an explicit calculation of contour integrals, or by a self-consistent assumption, 
the integral on the complex line (the standard definition of the Fourier transform) might be 
considered to be rotated onto the real line so that real valued conjugate parameters can be 
considered. 

The scaling of the Fourier transform by A" reflects an assumption of extensive entropy 
and is also a necessary feature in scalable solutions of the saddle-point method. With the 
specific choice the order parameter <3> is normalized. The choice of an additional factor C in 
the sparse order parameter definition is chosen so that <E> and <3> are normalized. 

The trace over replicated spins is finally taken to give an expression for free energy (17), 
composed of terms (18), (20) and (Q 2 ) up to ensemble dependent constant terms and 0(1/N) 
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corrections. The Q2 term is (19) in the Poissonian variable connectivity, and in general 

& = -log/^(<l(5)) C/ expJ^g a ^+ Qia^S^S"* l\ , (A20) 

\ S { a («1.«2> j / c/ 

where the average over c/ is with respect to the marginal variable connectivity distribution, 
uniform or Poissonian. 



1. Modifications to the saddle-point equations 

The saddle-point equations can be written down for the general case (A20), the general- 
ization of (21) in the sparse order parameter is 

$(<x) oc U f [$(<7)] C/-1 exp \ J2^ aa+ E Q{*u*2)° ai v a2 \) ■ (A21) 

\ ^ a (0:1,0:2) 

The dense order parameters are determined through the recursions 



q a = J2 a <* V ^ 5 <H°w) = E ° ai ° a2 n*) I (A22) 
(T a 



subject to a normalized distribution 
V(S) = 



$(5)j C/ exp l£,q a S a + J2 ?(oi,a 2 )^ Q1 ^ 2 [ ) • (A23) 

[ o (01,02) J / 

The conjugate saddle-point equations are unchanged in form (23). 
APPENDIX B: ORDER PARAMETER CONSIDERATIONS 

An interpretation for some parameters can be gained by consideration of derivatives of 
the free energy with respect to (5 and simple random external fields z. This may also be used 
to prove the consistency of some method assumptions in the case of replica symmetry. The 
choice of a random field is primarily to allow a concise inclusion within the replica method 
equations. It is equivalent to working directly with fields that are conjugate to quantities 
such as T i T j-> or w hh annealed random fields in some cases. 

A derivative of fife, with respect to (5 determines the energy density. It is well known the 
entropy becomes negative in both the VB and SK models at low temperature if incorrect 
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symmetry assumptions are used, this is also observed in some of the systems presented. 
Consider also a perturbation on the Hamiltonian to include a non-zero external field 

ah(t) = ^2 z ^ ; z i = zr li 5 ( B1 ) 

i 

where rji is either uniform or randomly sampled from a symmetric distribution, and z is small 
and positive. In the case that rji is uniform, the derivative of the free energy density with 
respect to z in the limit of small z is easy to evaluate and gives the magnetization, which is 
coincident with the sum over first replica moments at the saddle-point in the self-averaging 

case 

££ = ^£fo> ■ (B2) 

a i 

The derivative when ff = {1,-1} is coincident with the susceptibility and also the sum over 
2-spin correlations at the saddle-point 

d 



d 
On 
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£ W> = ^ £ fe> - <T^ <t,> . (B3) 

n=0 {a u a 2 ) (ij) 



These are useful quantities in evaluating the emergence of ferromagnetic order, and in de- 
termining phase transitions. 



1. Assumption of real valued integration variables 

An assumption of the saddle-point method used to evaluate the exponential term describ- 
ing the free energy is that only real valued integration parameters (order-parameters) need 
be considered. This Appendix demonstrates that any physical solution must be real valued 
in its first two moments, this is assumed to extend to higher order moments in the sparse 
order parameter $. 

A useful variation on (Bl) for purposes of general analysis are identified by a class of fields 
aligned with interactions. Consider the factor graph representation of the Hamiltonian (A4). 
Each of O(N) sparse interactions also has a unique label fi = (ij), with other ordered edges 
(ij) subject to weak (dense) interactions. Therefore 

* = (* D + zSj h) + £ Cw) ( z °Jm + zS 4,)) > ( B4 ) 
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is well defined and includes a component dependent on the sparse substructure and one 
on the dense substructure. Each of are assumed to be (a default for discussions), 

uniform (1), or quenched variables sampled independently from {—1, 1}, with {z s , z D } being 
infinitesimal real positive fields. Unordered matrices are used in (B4) to describe their 
ordered counterparts, so that is (ij) or (ji) as ordering dictates, for each ordered pair 
only one quenched parameter exists. 

Physical interpretation for these fields is as follows, whenever J ^ and rjk = I deriva- 
tives with respect to z D give a magnetization. When i] k = { — 1, 1} susceptibilities such as 
(B3) are determined. When = 1 the derivative probes alignments of variables with 
couplings, again giving a measure that can distinguish an ordered phase from a param- 
agnetic one. The more complicated physical quantities involve a quenched random field 
= { — 1, 1}, for example 



d [(3{z D ) 2 } 



identifying a type of susceptibility. 

The free energy in the replica formulation, with inclusion of these infinitesimal fields 
involves a modification of the factor-centric (Qi) term (18) in the free energy. Following 
Appendix A 

& = - E« ¥Mq a + 2^ <C<«>»?« - §E W)Q2 > P 2 J 2 (?<«!,«,> + 2(^) 2 <(C fo -)) 2 » <Z(« llQ2 ) 

¥ lo gE55' $ ( 5 ) $ ( 5 0/^^)<exp{-^Ej^ + ^Cfo))5 to }> ., • 

(B6) 

The variable-centric term is also modified from (A20), following from Appendix A the av- 
erages over are straightforward, but the average over is involved for the general 
case, requiring additional order parameter definitions. Consider the case that J^^ = J s is 
uniform for simplicity. The expression then becomes 



^ = -iog(E5( <|) ( 5 )) C/ex p{E a ^^+E (ai , a2 )^ 1 , Q2 >^ 1 ^ 2 } 



x exp {(3r) k (c f z s + ( J + AJ)^)£ a 5 a }> c/ % 



(B7) 



with A a normally distributed parameter. 

— * 

Quantities derived through the proposed choices for {(, if} are necessarily real- valued at 
a saddle-point and may be calculated in the replica framework given the self-averaged free 
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energy. The derivatives with respect to C M are particularly transparent, for example taking 
?7(jj) to be a quenched parameter from ±1, the derivative at the saddle-point gives 



d 



d 



fe 



d_ 

dn 



n=0 



j 2 £ "U, 

{"1,02) 



(B8) 

must be real-valued. In the 



so that whenever J is non-zero the RS term Y\ „ s ql \ 
case J 2 = the order parameter definition is in any case redundant and can be removed 
from the free energy. A derivative with respect to z s can produce a similar constraint on 
the second moment of the sparse distribution [25]. Taking rji to be uniform demonstrates 
that first moments of the order parameter must also be real. 



2. Spin glass susceptibilities 

The quantity probed through the BP stability analysis is a form of spin glass suscepti- 
bility. Consider two sets of random variables described by a joint probability distribution 
determined by identical quenched disorder except in a weak field term 

P(a,r) = ^exp^-H(a) -H(r) + zJ2vk(z 1 a l + z 2 a,)^ . (B9) 

where -21,-22 and rj k are quenched parameters sampled independently from {—1, 1}, and z is 
a small external field. Assuming self-averaging the limit ,2^0 ought to smoothly recover 
the equilibrium description of a single system (as evaluated using an RS assumption for 
example). An expansion of the self-averaged free energy finds at 0(z 2 ) and 0(z A ) constant 
terms, and terms dependent on the macroscopic magnetization and susceptibility. These 
terms are expected to be well defined in the small z limit if the population dynamics method 
converges, since they are coincident with the RS order parameters moments at the saddle- 
point, and stability is tested by fluctuations implicit in population dynamics. However, at 
0(z 4 ) there is also a dependence on spin-glass susceptibility 

*SG= ^E(^')-^)^')) 2 • ( B1 °) 

The spin glass susceptibility, as opposed to linear susceptibility (B3), probes a symmetric 
local instability, symmetries which might allow the linear susceptibility to be convergent are 
absent. In the replica formulation non- convergence of the spin glass-susceptibility in the 
limit of small z provides a sufficient criteria for failure of the RS assumption. 
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Non- convergence can be tested by a local stability analysis of the order parameters in a 
single (uncoupled) model under iteration of the saddle-point equations. The form of spin 
glass susceptibility tested in the BP framework is not exactly (BIO), one must consider an 
external field reweighed by interaction strengths to determine an appropriately reweighed 
set of perturbations (B4). The site factorization is then only achieved by definition of new 
order parameters, and it is the linear stability of the original order parameters towards this 
new description at z = that is a sufficient test of divergence, and comparable to stability of 
BP fields under iteration. However, asymptotic divergence of the BP equations as presented 
is expected to be a sufficient criteria for divergence of the standard spin-glass susceptibility 
in the equilibrium analysis, accurate as a predictor of trends and parameter dependence. 
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